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ABSTRACT 

A total of 359 retained seventh and eighth grade 
students from the Austin (Texast Independent school District*s junior 
high schools participated in a sumner enrichment program that ifas 
designed to improve their basic skills and decision making skills and 
tc provide them ifith a successful school experience. Diagnostic 
information ifas used to assign students to reading and mathematics , 
classesr but assignment to decision making classes ifas done on a 
random basis* The academic classes ifere individualized as much as 
possible^ ifhiXe the decision making classes involved lectures^ filmsr 
and student participation in decision making activities* pre and 
posttest data included the reading comprehension^ math concepts and 
problems^ and math computation subtests of the loifa Tests of Basic 
Skills (ITBS) ^ ifriting samples^ and scores on a decision making test* 
The results shoved that the progtam*s objective of producing gains in 
6QH of the. students ifas met on the math computation subtest of the 
XTBS and in writing ability^ but not on the remaining subtests of the 
ITBS nor on the decision making test* The findings also revealed that 
seventh grade students shoifed significant improvement only on the 
math computation subtest^ trhile eighth grade student's improved 
significantly on all three subtests* (EL) 
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ESAA BASIC SUMMER SCHOOL REPORT 



INTRODUCTION 



The ESAA Basic Summer Enrichment Program was a cooperative program between ' 
Austin Independent School Dlstrlct*s Department of Secondary Education and 
the Region Kill Educational Service C^ter. The purpose of the Sumner 
Enrichment Program was to provide seventh*- and eighth-grade retalnees with 
a successful school experience and to improve significantly their academic 
and declslon**inaKing skills. Mare generally* the pur|>ose of the Summer 
Enrichment Program was to increase the probability that these students 
would succeed during the 1980*81 school year. 

Program students were selected from lists of retalnees obtained from each 
of AISD^s junior high schools. The program was conducted at one site 
(Martin Junior High School) with a total ^rollment of about 339 students. 

The Summer Enrichment Program had two objectives: 

1. To improve basic skills (reading* math* and writing) 

2. To improve declsion**malcing skills. 

Diagnostic Information was used in assigning students to reading and mathe- 
matics classes* but asslgnm^t to writing/decision-making classes was done 
on a random basis. Classes were small (N * 10^13) and were oriented toward 
active participation of students and strong individual attention. Academic 
programs were individualized as much as possible* based on the students* 
past performance. 

The decision-^Klng curriculum was based on the assumption that the types of 
students who are found in programs such as th^ Summer Enrichment Program are 
likely tb^^adhere to a highly deterministic philosophy .vis-^a^is the problems 
that they face on a dally basis. Many of these students believe that they 
can contribute very little to the solution of their problems. The curricu- 
lum was aimed toward changing students* attitudes toward decision making* 
as well as toward developing decision^making skills. Activities included 
listening to lectures about decision-making skills* watching films* and 
participating in decision-snaking exercises. 

The program took place during June of 1980 for 3% hours a day. 

PABT I 

ACHIEVEMENT TEST ANALYSES 

Aho^^w OjJ thz ackizvmwt tut n^^atts OjJ tkz SwmnA. En/Ucimmt ?Kognam 
^tadrnts Kzvzalzd thz ^oZtoiMng ^Jjfidbig^^t 

1) Thz obj^ctLvz that 60% o(J thz 6tadmt6 matd 4^hovo po6^vz 

pnz- to po6ttt&t gaifa vooA meX, only on ikz M(tdi CompvJjoMjon ^aWwt. 
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2) Stati^tiaUlij ^^ignliicant p^e- to po6ttt&t g<iin6 occuA^ed on atl x 

3) Seuejttfe gmdM^ ^homd ^tat^ticaZty -ixgn^iJ-cautt -unpnjovmznt only 
on iht ttautk CompuXcuUon 6abtutt wrfuXe aJ^hpi gwdM^ ^pwv^ 
^^igni^cantly on aZl thKzt ^uWeii^. 



Purpose 

Results from the Iowa Tests of Basic Skills (ITBS) were studied In order 
to collect Information relevant to the following questions: 

Decision Question Dl : Should the ESAA Basic Summer Enrichment Program 
be continued as Is, modified, or discontinued? 

Evaluation Question Dl*l : Have the objectives for the Summer 
Enrichment Program be^ met for the 1930 funding period? 

Outcome Objective Dl*la : At the conclusion of the Summer Enrich- 
ment Program, 60 percent of the students will show improvement 
In academic achievement, as measured by the following ITBS 
subtests; Reading Comprehension, Math Concepts and Problems, 
and Hath Computation. 



Procedure 

Testing 

All junior high students were administered the ITBS, Form 7, in April, 1980, 
as part of the District's regular testing program. Seventh graders we^e 
administered Level 13, and eighth graders. Level 14. Scores from this 
administration were used as the pretest measure for students participating 
in the Summer Enrichment Program. More complete infonnatlon concerning 
this test administration can be obtained from the 1979-80 Systemwide Testing 
Technical Report (Publication No. 79.14). 

On June 30, 1980, the last day of the Summer Enrichment Program, the pro- 
gram's math and reading teachers administered the Reading Comprehension 
subtest, the Math Concepts and Problems subtest r and the Math Computation 
subtest of the ITBS (Form 7). The teachers were provided with adapted 
instructions from the ITBS manual, as well as a checklist, specifying the 
procedures to be followed in administering the ITBS. Observations of the 
testing procedures in two randomly selected classrooms suggested that, due to 
time constraints, some of the math teachers may have had to shorten the 
allotted testing time for the Math Concepts and Problems subtest. Results of 
this subtest need to be interpreted cautiously in view of this observation. 
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Since the reading and math classes were not completely segregated by grade, 
special attempts were made to Insure that the students were administered 
the same levels of the UBS during the pre- and posttest sessions. Two 
or three likstances occurred during the summer school testing, however, in 
which seventh graders were administered Level 14, and eighth graders were 
administered Level 13. 

Ik) makeup testing session was conducted £or the Suraoer Enrichment Program. 
Data Analyses 

Pretest ITBS raw scores were obtained from the master printout for district*-^ 
vide testing. ITBS grade equivalent scores were obtained from the ITBS manual 
and verified. The posttest scores were obtained by machine-^scoring the answer 
sheets, using an adapted version of the computer program that was used to 
score the answer sheets for the 1979*80 districtvide testing program. Pre* 
and posttest gr^de equivalent scores were written out on data sheets, with 
student ID and grade coded. The data were keypunched and verified at U. T. 
The card file layout is shown in Figure X. 

Results were analyzed by means of SPSS computer libraries at U. T. Frequency 
distributions were computed to determine the number and percentage of students, 
by grade, making positive gains (greater than zero) on each subtest, based on 
grade equivalent scores. T**tests were calculated to determine the statistical 
significance of the pre* to posttest gains on each subtest^ by grade. Scores 
for a total of 146 seventh graders and 171 eighth graders were used in these 
analyses. Only students for whom both pre- and posttest scores were available 
were included. 



Results 

Obi active t Sixty percent of the students attending the Summer Enrichment 
Program will show improvement in academic achievement. 

The objective was met for the Math Computation subtest^ but not for the 
Beading Comprehension or Hath Concepts and Problems subtests. Figure 2 shows 
the percent of students making positive gains (greater than zero), by grade 
and Subtest. Improvement was shown by 73.3 percent of the students in Hath 
Computation^ by 36.6 percent of the stud^ts in Hath Concepts and Problems, 
and by 32.8 percent of the students in Reading Comprehension. Eighth graders 
met the objective In both Hath Computation and Reading Comprehension^ but 
seventh graders met it only in Math Computation. 
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FILE ID / / CARD FILE UYOUT . LOCATION: 

COHH^S ITBS Girade Equivalent Scores, ESAA Basic Summer Enrichment Program 



FORMAT FOR ANALYSES 



Field 


Columns 


Description , 




1—7 






-8 






9-11 


Pre 1 fR^Aillnft CoTfiD (mtA Rniilv ^ 

-L. , 




12-14 


" 2 (Math Conceots " " ^ 




15-17 


" 3 ' (Math Comouta. " " ^ 






jrost ± (Keaaing comp. bra. cqulv*) 




21-23 


• • 

" 2 (Math CancfiDtA " " ^ 




24-26 


" 3 (Math C6tooiita m ii \ 







































Figure I. GRADE EQUIVALENT SCORES. ESAA BASIC SUMMER ENRICHMENT PROGRAM. 
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SUBTEST 


7th 
(N " 146) 


8tb 
(N " 171) 


Total 
(N " 318) 


Reading 
Conprehenslon 


50.0 


62.6 


56.6 


Hath Concepts 
and Problems 


49.3 


56.1 


52.8 


Hath 

Cooputatlon 


71.9 . 


74.9 


73.3 



Flgure.2. FERCEHTAGE OF STUDEHTS SHOWING POSITIVE 
GAINS (GREATER THAN ZERO) IN ITBS GRADE 
EQUIVALENT SCORES. 
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Figure 3 shows the tests of significance of pre- to posttest gains, by 
grade and subtest. Seventh graders shoved significant tmproveoent only 
on the Math Computation subtest, whereas eighth graders, as well as the 
total group, improved significantly on all three subtests. 

A comparison of the above results with those of the 1979 Summer Enrichment 
Program reveals two comonalities. First, in both cases, the program's 
impact was more evident on eighth graders' achievement than on seventh 
graders' achievement. Eighth graders met objectives and showed significant 
gains on more subtests than seventh graders did. Second, both programs 
seemed to impact more on Math Computation scores than on either Reading Compre-^ 
hension or Math Concepts and Froblems scores. Objectives were met and signifi* 
cant gains occurred more frequently on the H&th Computation subtest than on 
the latter two subtests* In general, eighth graders tended to improve on 
all subtests, while seventh graders tended to improve only on the Math 
Computation subtest. 



PART II 
WRITING SAMPLE ANALTSFS 



Atuity^t& 0^ ^tudojftts^ mlUng ^ajnpJU& n^zvMJUd ihz ^oZlovUng: 

1) Thz obj^jntivz that 60 pe^cejtt tkz pnogfum 6tadwt6 mold 
intp^vz in mcUng ablUty m& met* 

2) A significant pnz- to postt^t gain accused in miting 
ability. 



Purpose 

Writing samples were obtained from the students to provide information 
relevant to the following questions: 

Decision Question Dl : Should tL ESAA Basic Summer Enrichment Program 
be continued as Is, modified, or discontinued? 

Evaluation Question Dl-1 : Have the objectives for the 1980 
funding period been met? 

Outcome Objective Dl-lb : Sixty percent of the students attending 
the Sunner Enrichment Program will show improvement in writing 
ability measured by a locally developed writing instrument. 
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SEVENTH GRADE (N " 146) 



SUBTEST 


PSETEST MEAN 
GRADE EQUIVALENT 


POSTTEST MEAN 
GRADE EQUIVALENT 


T-RATIO 


SIGNIF 


Reading 
Conprehensioa 


6.1 


6.1 


.55 


ns 


Hath Concepts 
and Problems 


6.3 


6.4 


1.25 


ns 


Hath Computation 


6.3 


6.9 


7.35 




EIGHTH GRADE (N - 171) 


SUBTEST 


PRETEST MEAN 
GRADE EQUIVALENT 


POSTTEST MEAN 
GRADE EQUIVALENT 


T-RATIO 


SICNIF_ 


Reading 
Conprehension 


6.6 


6.9 


2.77 


A 


Hath Concepts 
and Problems 


7.0 


7.2 


1.61 


A 


Hath Computation 


7.0 


7.8 


9.52 


A 


COMBINED GROUP (N - 138) 


SUBTEST 


PRETEST MEAN 
GRADE EQUIVALENT 


POSTTEST MEAN 
GRADE EQUIVALENT 


T-RATIO 


SIGNIF 


Reading 
Comprehension 


6.3 


6.S 


2.37 


A 


Hath Concepts 
and Problems 


6.6 


6.8 


2.20 


A 


Hath Computation 


6.6 


7.3 


11.97 


A 



Figure 3. T-TESTS OF THE DIFFEREKCE BETWEEN PEE- AND POSTTEST ITBS 
GRADE EQUIVALENT SCORES BY GRADE. 

^Refers to a significant diCCereace at the .05 level of probability, 
as refers to a noa^ignificant difference at the -OS level of probability. 

er!c 7 
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Procedure 

Instrument Description 

The writing instrument was developed to aid in evaluating both the writing 
and the affective components of the Suamier Enrichment Program. Beglon XIIX 
staff developed The writing stimulus, which Is shown In Attachuent 1- The 
writing stiiauXus will be discussed In greater detail in the section deal- 
ing with affective measures. 

The scoring procedures for the jwrltlng sample were developed by a Project 
Specialist from MSD's Department of Secondary Education, with technical 
assistance from ORE staff. The Instrument was specifically geared toward . 
measuring progress in the writing of very "basic" writers, who are typically 
found In programs such as the Summer Enrichment Program. The scoring form 
Is shown In Attachment 2. Individual Items reflect the objectives toward 
which writing instruction was directed In the Sumner Enrichment Program. 
'The Instrument taps such areas as word fluency, spelling, sentence Indl* 
cators, usage, paragraph development and writing content. The Scoring 
Manual, which was developed by the Project Specialist from Secondary 
Education, explains the rationale and procedures for scoring the individual 
Items In detail. It Is shown In Attachment 3. 

Testing 

The writing teachers obtained the writing samples (along with an affective 
Instrument to be described In Part III) In the writing classes on the first 
and last days of the Summer Enrichment Program. Because no alternate form 
of the writing stimulus was available, half of the students were assigned 
to a pretest group, and half to a posttest group. Students were paired 
within classes on the basis of sex and ethnicity, with members of each 
pair being randomly assigned to either the pretest or posttest ^oup. 
For students who had no match In a given class, the following matching 
procedures were utilized: (1) if possible, they were matched on the 
basis of sex; If not, (2) an attempt was made to match them ou ethnicity; 
and If neither of these two options was possible, (3) they were matched 
randomly with other unmatched students In the class. "Leftover" students 
in classes containing an odd number of students were eliminated from the 
analyses. Lists of the students to be pre* and posttested were given to 
the writing teachers before the first and last days of class* along with 
brief instructions for administering the instruments and checklists remind** 
ing the teachers of the testing procedures. 

CodinR the Writing Samples 

Four Ph.D. students frcm U. T. with strong backgrounds in English grammar 
and sentence structure were hired to rate the writing samples, using the 
locally developed instrument* A two-hour training session was conducted by 
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the Froject Specialist from Secondary Education oa Wednesday^ June 25. 
Training consisted of explanation of and practice with the scoring procedures » 
using writing samples obtained from students who were not in the Suoimer En- 
richment Program. 

Originally » It was planned to have the coders rate the pre- and posttests 
together^ without telling them which essays were pretests and which were 
posttests. When It was learned^ however^ that the project*s funds could 
not be spent after June 30» this plan was altered. The pretests were 
coded on Thursday^ June 26» and Friday^ June 27» while the posttests were 
coded on the afternoon of Handay» June 30» as soon as they were collected 
from the writing teachers. An interrater reliability study was conducted 
by having each coder recode approximately 12 pre* and 12 posttest writing 
samples that had been previously coded by another coder. The four coders 
spent a total of approximately 40 hours rating more than 400 writing samples. 

Data Analyses 

Because of limited funding resources » It was not possible to analyze pre- 
to posttest gains on the individual Items of the writing instrument. 
Instead^ a total score was derived from the writing sample^ and pre-^ to 
posttest gains were calculated on It. The total score was obtained by 
treating eadh Item as a two» three» or five-point rating scale» with zero 
as the origin. For the first five Items » which were coded In terms of raw 
frequencies and percentages » arbitrary scales were used to transform them 
Into rating scales. In order to maximize the sensitivity of each of the 
first five Items » a rough estimate of the range of scores on each Item 
was obtained and used as e guide for establishing the rating scales. 

For Item IA» which measures word fluency^ It was determined^ after examining 
20-30 randomly selected writing samples » that the samples rarely contained 
more than 60 words. Subsequently » the rating scale that was adopted in* 
volved assigning a score of zero to writing samples containing 1-lS words » 
a score of one for 16-^30 words » and so on» up to a score of four for 61 or 
more words. Similarly^ for Item IA2»^ which measures the percent of words 
spelled correctly^ It was seen that few students spelled less than 30 per- 
cent of the words correctly. Thus» a score of zero was assigned to 
writing samples showing 0*80 percent of the words spelled correctly^ a 
score of one for 81*85 percent spelled correctly^ and so forth. For 
Items IB1-IB3» which measure the percent of sentence Indicators Included 
In the writing sample^ relative to the potential number of sentence Indi- 
cators » It Is possible to obtain percentages that are greater than 100. 
Since 100 percent Is the Ideal score for these Items » a score of four was 
assigned to writing samples showing 81-120 percent of the potential sen- 
tence Indicators » a score of three for both 61-80 percent and 121*140 
percent etc. 
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The pre* and posttest mrltlng samples were collated according to the lists 
of matched student pairs and were keypunched and verified at U. T* Only 
pairs of students for whom both pre- and posttests were available were 
Included In the analyses. The data were keypunched directly from the 
scoring forms» with the exception of the first five ltems» which were 
manually recoded before they were keypunched. The card file layout Is 
shown In Figure 4. 

The Individual Items were prioritized according to how much weight the 
Project Specialist thought they should carry In determining the total 
score » and were weighted In the computer programs. In Attachment 2» the 
numbers corresponding to the choices for the last 10 Items show how the 
Items were weighted. 

SPSS computer libraries at U. T. were used to analyze the writing sample 
data. Interrater reliability was examined by calculating Spearman correla* 
tlon coefficients on the Individual Items » and Pearson Product Moment 
correlation coefficients on the total writing scores. The protocols that 
were Included In the reliability study were recollated and keypunched 
separately^ using a similar format to the one In Figure 4. A frequency 
distribution was calculated to determine the number and percentage of 
matched student pairs shoving positive pre^ to posttest gains (greater than 
zero) on the total writing scores. A t-test was 'calculated to determine 
whether the mean pre- to posttest gain In total writing scores was sig- 
nificantly greater than zero. Only pairs of writing samples for which 
both pretests and posttests were available were included In the analyses 
(N - 139). 



Results 

Figure S shows the results of the Interrellablllty study. The pre- and 
posttest reliability coefficients of .773 and .865 show that the writing 
Instrument Is adequately reliable and that the reliability seemed to 
Increase as the coders gained coding experience from pre- to posttest. 
Items of low reliability Included II (usage) » II31 (presence of top.^ . 
sentence within paragraph), IIB2 (presence of "ejcpanslon** sentences), and B 
(giving Information, Ideas or details called for In stimulus). 

The frequency distribution showed that 66.9 percent of the posttest 
students showed higher writing sample scores than those of matched pretest 
students. This finding Indicated that the objective that 60 percent of 
the students would show positive pre* to posttest gains In writing sample 
scores was met. The t-test showed that the posttest writing scores were 
significantly higher than the pretest scores, at the .0005 level. The 
mean pretest writing score was 28.88, whereas the mean posttest score was 
38.47 (n - 139). The t*ratlo was 5.88. 
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FILE IB / / CARD FILE lAYOUT MXyiTIONt 

.comment s: writimg sample, esaa basic sumher enrichment program 

1 format for keypuncher 



Field 


Columns 


Description > 




1 ^ 11 


PRETEST STUDENT'S (FIRST STUDENT'S) LAST NAME 




12 - 19 


II II II II n^nniBa ^Vi^hCn 

" " " FIRST NAME 






SCORES FOR PRETEST STUDENT (FIRST STUDENT) 


• 




IN COLUMNS 20 - 34 




20 


ITEM lA 61^4, 46-60*3 > 31-45=2 > 16-30=1 > 0-15-0 




21 


' IA2 96-100Z"4 , 91-95Z"3 » 86-90%"2 , 81-85Z"1 » 0-80Z"0 




22 


121-140%-3» 141-160Z»2» 161-180%-1» 181Z-0 
" IBl 81-120%-4. 61-80Z-3. 41-60%«2. 21-40Z-1. 0-20%-0 




23 


" IB2 " " " " " 




24 


" IB3 * " " " " " 




25 


" II GENERALLY STANDARD«2» SOME LAPSES-1» FREQUENT LAPSES-0 




26 


" IIAl YES • 1» HO - 0 




27 


" IIA2 " " ■ 




28 


" IIBl " 




29 


" IIB2 2 or t » 2» 1 = 1» 0 = 0 



Figure 4. WRITING SAMPLE ANALYSIS FORMAT. (Page I of 3.) 



Pagel of 3 
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FIU ID / / CARD PILE LAYOUT • LOCATION t 

mWfRNTt WRITING SAMPLE» ESAA BASIC SUMMER ENRICHMENT PROGRAM 



FORMAT FOR KEYPUNCHER 



17 



Field 


Coluaas 


Description • 




30 


ITEM IIB3 2 or -1- « 2» 1 - 1» 0 = 0 




31 


" IIM YES " 1» HO = 0 





32 


" A TO A GREAT EXTENT » 2» TO SOMEEXTENT - 1» HOT AT ALL - 0 




33 


ft g It M M 




34 


99 99 fl 




35 - 45 


POSTXEST STUDENT^S (SECOND STUDENT'S) LAST NAME 




45 - 53 


II II II ll nnm %i m. un 

' FIRST NAME 






cr^pvc nfVA i>acttpct cninvuT /ccpnun c'^fm7uT^ tu pniiiMiiQ SA ^ fA 

aLiUfvEO I^UK irU&i.iC9i diUUIVDIi ^dtibUnU diUiPEIVl/ In ljUliUnW> J*v oo 




54 


ITEM lA 61+.- 4» 46-60 - 3 » 31-45''2» 16-30 «• 1» 0 - 15 « 0 




- 55 


" IA2 96^100%»4» 91 - 95% - 3» 86 - 90% - 2» 81 - 85% - 1» 0 - 80% - 0 






121 - 140% - 3» 141 - 160% - 2» 161 - 180% - 1, 120% + - 0 




56 


" IBl 81-120% » 4» 61 - 80% - 3» 41 - 60% « 2» 21 - 40% » 1» 0 - 20% - 0 




57 


" IB2 " " " " 




58 


" IB3 '* '* '* " 



Figure 4. (Page 2 of 30 
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Page 3 of 3 

FIU ID / / CARD HLE LAYOyT . LOCATION t 

COMMENTS WRITING SAMPl£» ESAA BASIC SUMMER ENRICHMENT PROGRAM 



Field 


Coluana 


Descriptlolt 




59 


ITEM II GENEBALLY STAilDARD > 2» SOME LAPSES « 1» FREQUENT LAPSES 0 




60 


' IIAl YES ■ 1> HO ■ 0 




61 


II TTA^ II tl 

IIA2 " 




62 


>l TTBl It II 

IIBl 




63 


IIB2 2 or + « 2> 1 « 1> 0=0 






>t TTno II II II 






xxDH ZED * x» nu u 




66 


*' A TO A GREAT EXTENT ■ 2. TO SOME E3CTEHT ■ 1. MOT AT ALL ■ 0 




67 


II g II II II 




68 


11 Q « II II II 



























Figure 4. (Page 3 of 3.) 



20 



80.19 





PRETEST 


(N » 45) 


POSTTEST 


(N • 49) 


TOTAL 


(N - 94) 


ITEM 


R 


SlKQlf 


R 


Slgnlf 


R 


Slsnlf 


lA 


.985 


.001 


.982 


.001 


.986 


.001 


IA2 


.740 


.001 


.812 


.001 


.770 


.001 


IBl 


.592 


.001 


.560 


.001 


.575 


.001 


IB2 


.647 


.001 


.530 


.001 


.590 


.001 


IB3 


.637 


.001 


.659 


.001 


.646 


.001 


II 


.415 


.003 


.264 


.034 


.330 


.001 


IIAl 


.915 


.001 


.948 


.001 


.932 


.001 


ITA2 


.217 


.077 


.711 


.001 


.434 


.001 


IIBl 


.247 


.052 


.122 


.203 


.212 


.021 


IIB2 


.309 


.022 


.313 


.015 


.363 


.001 


IIB3 


.419 


.003 


.676 


.001 


.570 


.001 


IIB4 


.482 


• .001 


.802 


.001 


.674 


.001 


A 


.729 


.001 


.536 


.001 


.639 


.001 


B 


.423 • 


.002 


.356 


.007 


.390 


.001 


C 


.912 


.001 


.949 


.001 


.930 


.001 


WRITING 














TOTAL 


.773 


.001 


.865 


.001 


.838 


.001 



Figure 5. INTERRATER RELIABILITlf COEFFICIENTS FOR WRITING 

SAMPLE: INDIVIDUAL ITEMS AND TOTAL WRITING SCORE. 
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PART III 

AHALTSES OF DECISION-MftS^IMG MEASURES 

1] -Tkz ob/ectcve that 60 pwcertt the Summe/t Ei/vUchmznt 
P^gfum ^tudwt6 muJtd 4jnp^o\fz in dzcU ion-making 
^kUti OKU not met. 

2) P/Le* to po^ttz^t ^ain^ in dzcX^n'makin3 ,^ki£JU waz 
not U^niiicjajfitty gn^catz/t than ztfio* 



Purpose 

the ESAA Basic Stmmier Enrichment Program tests of decisioniaafcing skills 
were administered to provide information relevant to the following questions: 

Decision Question pi : Should the ESAA Basic Sumner Enrichment 
Program be continued as is, modified, or discontinued? 

Evaluation Question Dl*l : Have the objectives been met for 
the funding period, 1980? 

Oatcome Objective Dl-lc : Sixty percent of the students will 
show gains in decision-making skills at the end of the Sumoier 
Enrichment Program, as measured by instruments designed to 
measure those skills. 



Procedure 

Instrumegts 

the Decision-Making Questionnaire and the Decision San^le were both developed 
by 0. R. E. staff, in coordination With the Project Staft, to measure the 
affective outcomes in the Summer Enrichment Program. The Decision-Making 
Questionnaire is an 18-item, five-point Cikert scale, that taps all of the 
projected student outcomes of the decision-^klng component of the Summer 
Enrichment Program. Some items were written to show attitudiaal changes 
toward decision making; e. g., always have chances to choose different 
ways to act,** and **I would rather make iny own decisions than let someone else 
make them for me,** whereas others were written to obtain a self-report measure 
of the students* skills vis-*a--vis the critical aspects of the decision-making 
process; e.g., '*When I have a problem, I can usually see two or three ways 
to handle it,'* and "I can imagine what might happen (both good and bad) when 
I decide.** The Decision-Making Questionnaire is shown in Attachment 4. 
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the Decision-Malciiig Sa«ple» vhicU la narrower in acope than the Deciaion- 
Maklng-Queationnaire» neasurea atudenta* attention and perforaance in 
critical aapects of the decision-wking proceaa» aa evidenced by their 
written reaponaes to a atinulus. For thia lnatrument» the aane atinulus that 
was uaed for the Writing Saaple waa utilized (aee Attachaent 1). The atlnu- 
lua begins with a paragraph^length deacription of a peraon who has a problen 
to aolve» followed by aeven multiple choice and ahort answer queationa» 
designed to orient the atudents' thinking toward the critical aapects of the 
decision-making proceaa. At the end» there are instructions to write a 
letter to the peraon who is deacribed in the atinulus^ in order to advise 
the peraon of the important thlnga that ahe ahould consider in making her 
decision. The Decision-Making Sanplcfs are analyzed using items auch aa 
"Number of alternative couraea of action mentioned and ''Proportion of 
altemativea and conaequencea oidorsed by the atudent that are likely to 
move Sally toward her goal of becoming a veterinarian.'* The Deciaion 
Sample Scoring Form is shown in Attachment 5. 

Tea ting 

The Decision-Making Questionnaires were adminiatered &y the writing teachera 
in the writing clasaea^ together with the writing aamplea on the first and 
last days of the Summer Enrichment Program. Srief directions were provided 
for both inatrumenta^ aa well aa checklists to remind the teachera of the 
testing procedures. Balf of the students were pretested and half were 
posttested. Assignments to the pretest and posttest categories were the same 
for the Decision-Making Questionnaires as for the writing samples. (See 
Part XXf Testing Section^ for an explanation of the assignment procedures.) 

Coding the Decision Sample 

Three matched pretest-posttest pairs of writing samples were randomly 
selected from each teacher's folder and coded for decision-making skills^ 
using the Declsion**Making Sample form. A total of 30 pre- posttest pairs 
were coded. 

Data Analysis 

The Decision-Making Questionnaires were keypunched and verified directly 
from the instruments at U. T. The card file layout Is shown In Figure 6. 
Only matched pairs for which both pretests and posttests were available were 
Included in the analyses. Several matched pairs were eliminated due to incom- 
plete responses. One hundred eleven pairs :i)f -Decision-4faking Questionnaires 
were included in the analyses. The analyses of the DecisiocrMaking samples 
were calculated by hand. 
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FIU ID 



COMMENTS 



/ /• CARP FILE LAYOUT . 'LOCATION: 

DECISION-MAKING QUESTIOKHAIItE » ESAA BASIC SUHHER ENRICHMENT PROGRAM 



FORMAT FOR KEYPUNCHER 



Field 


Colums 


Description • 




1 * 11 


PRETEST STUDENT'S (FIRST S1VDENT*S) LAST NAME 




12 * 19 


1 






SCORES FOR PRETEST STUDENT <FIRST STUDENT) IN COLlMfS 20 37 




20 


ITEH 1 STRONGLY AGREE * ^» STRONGLY DISAGREE * 1 






'* 2 reverse 




zz 


3 






•* 




24 






25 


" 6 ■ 




26 


" 7 * 




27 


*' 8 




28 


" 9 




29 


" 10 




30 


" IX 



Figure 6. DECISION-MAKIKG QUESTIONNiaRE ANALYSIS CARD FILE LAYOUT. (Page 1 of 3.) 
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Pag e '2 of 3 

/ / CARD FILE lAYOUT LOCATION: 

00 

o 

DECISION-HAKING QUESTIONNAIRE » ESAA BASIC SUMMER ENRICHMENT PROGRAM ' 



FORMAT FOR KEYPUNCHER 



Field 


Columns 


Description 




31 


Zten 12 reverse 




32 


13 






ft t A 




Oh 






35 


*' 16 




36 


^ : * -s ~ 

" 17 STRflNGLY AGREE * 5, STRONGLY DISAGREE » 1 




J/ 






38-48 


POSTTEST STUDENT'S (SECOND STUDENT'S) LAST NAME 




49-56 


" • " " " FIRST NAME 






SCORES FOR POSTTEST STODENT (SECOND STUDENT) IN COLUMNS 57 - 74 




57 


Item 1 STRONGLY AGREE - 5, STRONGLY DISAGREE - 1 




58 


2 reverse 




59 


" 3 




60 


" 4 



Figure 6. (Page 2 of 3.) 



FILE ID 

COMMENTS 



Fits ID / / CARD FILB UYOUT . LOCATION: 

QMHRHTS ' DBCZSI<m-HAKING QUESTlOtlNAIRE^ ESAA BASIC SIAMER ENRICHMENT PROGRAM 



FORMAT FOR KEIfPUNCHER 



-- 




uc8crxpcxoit « - 




ill 


Item S 






" 6 ' 




63 


/ 




64 


tt A 




65 


tt H 

" 9 




66 


tt 4 a 

" 10 






11 




oo 


reverse 




69 


" 13 ■ 




70 


" 14 * 




71 


" 15 




72 


"16 




73 






74 


" 18 



Figure 6. (Page 3 of 3.) 
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Because of limited funding resources » pre- to posttest gains could not be 
examined on the individual items in the decision-making instruments. Instead 
total scores were calculated on both instruments^ in order to obtain overall 
measures of decision-^making skills » as conceptualized by the Project staff* 
Analyses were calculated on the total scores. 

The Decision*Making Sample has some items that are rated on a three-point 
scale» and others (yes/no items) that are rated on a two*point scale* The 
different size scales reflect the differential weights that the items were 
designed to carry in determining the total score* 

For both instruments » frequency distributions were calculated to show the 
number and percentage of student pairs making positive pre* to posttest 
gains* lytests were calculated to determine the statistical significance 
of the differences between pre** and posttest scores. SPSS computer libraries 
were used to analyze the responses to the Decision-Making Questionnaires » and 
the analyses of the Decision Samples were hand calucXated* 

Results 

Objective ! At the conclusion of the Summer Enrichment Program^ sixty percent 
of the students will show positive gains in decision*maklng skills. 

The objective was not met for~either of the two instruments designed to 
measure decision*making skills* On the Decision-Making Questionnaire^ 
52*3 percent of the posttest students showed higher scores than their matched 
pretest counterparts. On the Decision*Making Sample^ 43*3 percent of the 
posttest students showed higher scores than the matched pretest students. 

t-tests showed that the differences between pre- and posttest scores on 
both instruments were not significant at the .05 level of probability* ^For 
the Decision*Making Questionnaire » the mean pr^ittest score was 66*94» and 
the mean posttest score was 68*07* The t-ratio was 1,16 (n « 111)* For the 
Dec Islon-Making Sample^ the mean pretest score was 3*6» and the mean posttest 
score was 4*2. The t-ratio was 1.40 (n « 30)*(See Figure 7*) 
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t-TEST BETWEEN FKE- AND FOSTTEST 
DECISION-MAKING SAHPI^ 

(1) D - .6 

(ED)2/N - 324/30 - 10.8 
Ed2 - 170 



- .6/ .428 - 1.403 ns 
(o^.05 , 29 - 1.699) 



Figure 7. DECISION-MAKING SAMPLE t-TESX. 
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Attachment I 

Head the followliig paragraph and darken the space beside tlie answer you (page I of 2) 
think is best. 

Sue is an eighth-grader \ibo loves animals and hopes to become a 
veterinarian (animal doctor). She joined the 4-H Club this year 
to be with other teen^ers i^jo like what she likes. Ihe club 
has been fun, and ^e has leamed a lot there about anijnals. Fbr 
a project. Sue decided to raise a calf. Mien people in the 
neighborfaood con^lained to the police. Sue was told that she 
could not keep a calf in the city because it was against the law. 

1. What is Sue's goal? 
( ) a.To raise a calf* 

( ) b.To be president of the 4-H Qub* 

( ) c.To be a veterinarian* 

( ) d.To have fri^ds who like animals* 

. 

2. What is Sue's problem? 

( ) a. Her neighbors are angiy. 

( ) b.Her calf can't be kept in the city. 

C ) c.Her parents don't like her animal. 

( ) d.It's hard for girls to become veterinarians. 

3. Which o£ the following actions might help Sue readi her goal? 
Nfcirk eveiy answer lAich wculd help. 

( ) a.Quit the 4-H Qub. 

( ) b.Try to find a place outside the city vhere she could teep 
and raise her calf. 

( ) c.Keep the calf and raise it in the city. 

( ) d.Ask her family to move to the countiy so that she could 
raise her calf. 

4. If Sue chooses to ask her fanily to move to the countiy, vdiat 
would MOST LIKELY be the FIRST problem she vrill face? 

( ) a. The family might have to live in a smaller house. 

( ) b.The calf might not win airy prizes in the livestock shew. 

( ) C.Sue's fanily might not want to move to the country. 

( ) d. Riding the bus a longer distance to school might make Sue 
tired at night. 
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5. If Sue chooses to keep the calf in the city, ^ch of the 
followiJig is MOST LIKELY to result? 

( ) a.Sue*s parents will get angry. 

( ) b.lhe neighbors will con^lain again. 

( ) c. Nothing. 

( ) d.Sue will have to quit the 4-H dub. 

6. If Sue chooses to try to find a place in^the camtry lAere she 
can ke^ the calf, which of the following will she be showing 
is ijnpoTtait to her? Mark every answer that is in^rtant to Sue. 

( ) a. Staying within the law. 

( ) b.Making her 4-H Club sponsor proud of her. 

( ) c.Spendii^ tijne in the oxmtry. 

( ) d.Ccinpletijqg her 4-H Cltib project. 
/ 

7. List 3 soirees of help that Sue could use in making this decision. 

a. 

b. 

c. ; 

8. Sometimes young people have a hard time making decisions 
because they don't know how" to think about problems. , 
Write a letter to Sue (at least a paragraph long) and 

'..help her decide what to do about her problem with the 
calf. Don't make the decision for her, but remind her 
of the things she should think aHout before she makes 
her decision. 
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Attaehnent 2 
(page 1 of 2) 



SmaX SCHOOL E7AIDATI0M 
Writing Saaplft Scoring Form 



Stud«nt's ^nm 




^rds and SwtancM 



£|UQb«r of nords (omit a, an, tha) 
I* Huabar of miMp«Xl«d vorda. 
^2* Z of vorda spallad corractlY 
STumbar of thought units 



3* 

II, Usaga 



Riabar of Initial capitals 
Moobar of and marks 
Mti^ar of correct and marks 



IB3 



Usa tbMM^ alaaants as* guldaa to Idanttifylng^lavals of usaga: 

1* Subjact-varb agraamant 

2, Tarb tansa/sa^uancs indicators 

3* Pronotm casa 

4* Pronoun agraaaanc 

5* Approprista form of oodlflars 

6, Approprista placaaant of modlflars 

7* Us* of nsgatlvaa 

8, Idiomatic naa of languaga 



A* Ganarally standard uaaga (I or 2 arrors) 

B, Soma lapsas in uaaga (3 or 4 arrors) 

C* Fraquant lapsas in uaaga (5 or mara errors) 



III* Taragrapbs 

A*. Paragraph fdrmat 

Indantfd flrse santanca 
2, Flacamant of santancas in 
continuous flow 
B* Paragrspb davalopmant 



7«- 



no 



2. 



3, 



4* 



Baragtaph iua dafinlta topic avitanca or a santanca that 
indlcataa cba subjact of wbst is to foUov* 

ya s , - . 4 . n o 0 

Nuabar of santancas «ltbln tha parmgrapb that could ba 
ccnsidarad axpsnslon of thm first santanca* 

X Cg - lr A I - ? * 0 0 



Ruabar of words* or phrmsas that sarva as transition £rott ona 
Idas to anothar* i or ^ 4 I ^ 0 n 

Parsgraph baa a concluding santanca chat has a sansa of 
finality or ^'wrapping up"* ye s* A - d o . q 



ir* 

II 
II 



IIAl 
IIA2 

mi 

IIB2 
II33 
104 



34 
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Attachment 2 
(l>age 2 of 2) 



CONTEST 



B. 



C. 



Do«s th%. studaot focus on tta* tufe pr*s«ated by ch* stJaulus? 
V* «x* dttflnlns "focus oa cssk^' «s follows: 

A, Otttcts wciclBS Co ttas auditncs tbsc Is spccfisd in cbs sclaulus. 
To * giftsc mrtmn t L To so»s sattsn t 2 Not ac sl l Q 

QItss isforastloa, dscails or id««s callsd for in chs stinulus. 
To a grsst Mrtm c & To 9om mxtma t 2_ Hoc ac al l 0 

tnettss within tta* f onst callsd for in ttas sdnulos Ca*<** latcsr 
'*to9f%. £lrs£. parsoa asrxatin, sctps in a procsss, ate. ) 
To a. grsac Tcm c 4 To sows «xt«nt_.i_ Hoc at al l o 
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Attachaent 3 
(Page 1 of 4) 



Scoring Manual 



I. Words and sentences 

A. Number of words (omit a^» an_» the» and I) 

1. Niunber of mlsspellTd words 

a. Count the word every time It is misspelled. 

b. When apostrophes are wrongly included or omitted » count 
this a spelling error. 

c. When there Is a doubt about whettier the error is in 
spelling or handwriting^ check context to see if 
questionable letter is formed correctly elsewhere. If 
so» it is a spelling error. 

d. If a word is wrongly capitalized^ do not count this a spelling 
•error. 

e. If a word is wrongly separated (e.g.» foot ball) or wrongly 
joined (e.g.» a1ot)» count this as one error. 

2. Percent of words speTTed correctly 

' This will be surprisingly high» but remember that even a 
score of 91% may be several standard deviations below the norm. 
. 6. Number of thought units 

A "thought-unit" is a measure designed to show the percentage of 
actual sentence indicators (i.e. » initial capitals and end marks) 
in relation to the potential number of sentence indicators. 

A thought-unit requires the scorer to "translate" non-sentence 
writing into possible, sentences that are gr«nmatically acceptable if 
not always stylistically preferable. Some such measure is needed for 
very basic writers to determine if they are making progress in their 
ability to indicate sentences. 

Thought-units are based on the assumption that students do» indeed» 
think in sentences but lack the knowledge^ desire» or both to use 
the conventions of written language to indicate these sentences. 
The scorer's task is to take the inchoate writing of basic writers 
and translate it into what would be acceptable sentences if the 
conventions were employed. It will always be a somewhat arbitrary 
unit since two readers might occasionally differ as to which ideas 
might best be coordinated or embedded. In dealing with such a diffi- 
cult kind of writing^ however^ it is probably better to have some 
tool for measuring progress than none at all. 

Consider a thought-unit as what would be a graimatically acceptable 
sentence if all the conventions had been observed. Sometimes a 
student will coordinate two ideas » and even though he might write 
them as two fragments joined by and or but t count this as a legiti- 
mate thought-unit because if he KacT written his ideas in two inde- 
pendent clauses joined by a ciMina and a coordinate conjunction^ the 
sentence would have»been gratimatically accpetable. The same is true 
if the student subordinates one idea to another. If they could be 
written as a granmatically acceptable sentence^ call this one thought 
unit. 
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Attachaent 3 
(l«ge 2 of 4) 



S(»net1mes students join unrelated ideas into one long sentence. 
Divide the string of ideas into possible thought units. Don't be 
overly concerned by the fact that someone else might divide the text 
differently. There Is no way for us to know what the writer "really" 
fneant. We know the conventions of sentence writing, and he doesn't; 
therefore, the scorer's translation Is about the best we can co. 

Here Is an example of trenslatlng the paragraph of a very basic 
writer. 



I hate school Because It soiry and Boring Snobby 
teachers and Stupid Work long hours and a waste 
of time you don't hardly learn nothing 



I. hate school because It [Is] sorry and boring. 
[There are] snobby teachers and [who give us] 
stupid work. [We go] long hours and [which are] 
a waste of time. You hardly ever learn anything. 



There are four grammatically acceptable thought units here. Wherever 
Ideas may be legitimately coordinated or subordinated, do so in your 
translation. 

1. Number of Initial capitals 

a. Disregard all other capitals. 

b. Sometimes students do not know the cursive form of the capital 
and will write the lower case form larger. Count this a 
capital letter. 

2. Number of end marks 

3. Number of correct end marks 

There Is a necessary distinction between D and £ because students 
sometimes use an end mark, but an Incorrect one. The use of an^ 
kind of end mark does show progress In the student's ability to 
Indicate sentences. The use of the appropriate end mark shows 
progress to another level. 

II. Usage 

For the sake of consistency, usage Is defined as eight cotranon elements. 
If there Is a question about the meaning of any of the standard terms 
included as "usage," please consult any level of Warrlner's English 
Gramnar and Composition (Harcourt, Brace, Jovanovlch). 

III. Paragraphs 

A. Paragraph format 

1. Indented first sentence (stftf-explanatory) 
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Attachment 3 
(page 3 of 4) 



Z. Placenent of sentences In continuous flow. 

The above item refers to the physical appearance of the sentences 
on the page. Some students have no concept of the way a para* 
graph 'looks, so they write each sentence beginning at the left 
margin. 
B. Paragraph development 

1. Paragraph has definite topic sentence or a sentence that indi- 
cates the subject of what is to follow. 

The topic sentence need not be formally stated. 

2. Number of sentences within the paragraph that could be considered 
an expansion of the first sentence. 

"Sentences" may be thought units. If the thought units are re* 
lated to the main idea or topic sentence in some recognizable way, 
count them as expansion or support. 

3. Number of words or phrases that serve as transition from one idea 
to another. 

"Transition** here refers to such common words and phrases as 
first, on the other hand, so^, also , another , etc. Transition 
should be counted between actual sentences or thought groups. 
Oo not consider sMa transitional device since it is quite 
often used by basK writers as an a11*purpose connector when they 
do not know how to begin and end sentences. 

4. Paragraph has a concluding sentence that has a sense of finality 
or "wrapping up." 

Many students write conclusions very naturally. Many, however, 
will simply stop writing or try to conclude by writing "The 
End." If a sentence or thought-unit suggests a summing up, an 
emphatic point, or scHne kind of closure, consider it a concluding 
sentence. 
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Attachnent 3 
(page 4 of 4) 



Content 

Does the student focus on the task presented by the stimulus? 

"Content** Is usually evaluated quite subjectively as "Interesting," "logical," 
or "weli'developed. It Is difficult to determine whether or not we have 
taught a child to be more Interesting or more logical, but we can be more 
objective In evaluating how well he has learned to respond to the writing task. 

A. Directs writing to the audience that Is specified In the stimulus. 

Ifthe writing stimulus does not state or ln^ly any audience other than the 
teacher or evaluator, this category Is not applicable. If an audience Is 
stated or strongly Implied, the student's writing should reflect some 
attempt to address the Interest, the ability level, and the content of the 
audience. Ideas, vocabulary, sentence structure, and even nouns of direct 
address will give clues to audience awareness. 

B. Gives Information, details, or ideas called for In the stimulus. 

..If the topic of the stimulus relates to why the student likes school, the 
reader would expect to find reasons or examples. If the stimulus called 

.for a description of "a place I'd like to be right now," the reader would 
expect to find some specific details about how things look or how the 
writer reacts to certain scenes, objects, or people. 

C. Writes within the format called for In the stimulus (e.g., letter, story, 
first person narrative, steps In a process, etc.) 

If a letter Is called for, the entire letter format need not be used. 
Give the student credit If there Is some evidence that the writing Is a 
letter, even though It be just a salutation, closing, or direct address 
to the specified audience. 

If the stimulus calls for no special format other than the standard 
paragraph form. Is the paragraph developed according to a type of 
development such as reasons/examples, comparison/contrast, cause/effect? 

This Is an Imperfect Instrument for evaluating growth In writing skills. One 
of Its values, however, may lie In the attempt to make explicit those areas that 
are unquestionably subjective. Another value may lie In the attempt to stipulate 
the meaning of terms so that scorers may have some consistent limitations to 
their subjective evaluations. 

It Is an attempt to evaluate writing progress at the most basic level with 
scores that may have some ImpHcat-fons for the kinds of Instruction needed at 
this level. 
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NAME: 



GRADE: 



SEX: Hale 



Female 



DATE: 



HEAD EACH STAXEHENT. THEN CHECK HOW MUCH TOO AGREE OR DISAGREE WITH 
THE STATEMEMT. 



I* I make decisions (both big and 
little) each day. 

2. There isn^t much that I can do 
about oy problems. 

3. I always have chances to choose 
different ways to act. 

4. I know where to get help for loy 
problems. 

3. I know the reasons for my decisions. 

6. I can name some of oy goals. 

7. I can name some W priorities 
(things that are Im^sortant to me). 

8. I can see how my goals and priorities 
help me make decisions. 

9. When I have a problem^ I can usually 
see two or three ways to handle it. 



STRONGLY 1 
AGREE 1 


AGREE 1 


HOT SURE 1 


DISAGREE 1 


>• fia ■ 
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10. I can iaagina what might happen (both 
good and bad) when I make a daclsion. 

11. It's helpfiiX Cor me to take my time 
when I maka a daclaion. 

12. I often put o££ making hard deciaions. 

13. I voiiXd rather make m^ own declaiona 
than let someone else make them for wt. 

14. I am good at making decisions. 

13. Decisions that are right for me may be 
wrong Cor someone else. 

16. It^s helpCiiX Cor me to consider all 
my * alternatives when I decide. 

17. I can understand why diCCerent people 
inake dif Cerent decisions. 

18. When I decide, I can live with the 
conaequencea. 



i 


AGREE 1 


HOT SURE 


DISAGREE 


STRONGLY 
DISAGREE 
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Attachment S 



Writing SttDplt Scoring Fom: 
Dtclslon MUcing AbUltits 



KaiK Grade 



U number of alternative courses of action mentioned* 
0 I 

2* Number of sources of help mentioned. 

0 I 



3* Total number of reasons- mentioned for all alternative courses of 
action. - 0 ^ I 2 



4* Total number of consequences that are predicted for all proposed 
alternatives* 

0 I " 2 



S. Mentions one or more of Sally's goals* 

no yes 



6* Associates proposed alternatives with underlying values* 

go yes 



7. Proportion of alternatives and consequences endorsed by the 'studentr 

Chat are likely to move' Sally toward her'goal of becoming a veterinarian* 
none of "^tfae m some of tbem____ all of them— — 

6* Discourages Sally from choosing alternatives end consequences that are . 
likely to move her away from her goal of becoming a veterinarian. 

no yes 
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